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There is widespread consensus that improving our na¬ 
tion's competitiveness in science fields urgently demands 
improved science, technology, engineering and math 
(STEM) education, particularly for underserved youth. As 
a result, policymakers, funders, and educators have led a 
call to stimulate the U.S. STEM pipeline. Recognizing that 
schools can't do it alone, they have called for "all hands 
on deck" to boost STEM achievement, ignite passions 
in science, and expose students—particularly female 
and minority students—to STEM career possibilities. 

Expanded learning opportunities, such as after¬ 
school and summer programs, are particularly well po¬ 
sitioned to help address the STEM education crisis (Af¬ 
terschool Alliance, 2011). A large percentage of youth 
participating in afterschool programs are members of 
groups traditionally underrepresented in STEM fields. 
Additionally, the nature of these programs—featuring 


low student-to-staff ratios and opportunities for hands- 
on and project-based learning—makes them an ideal 
environment for inquiry-based informal science edu¬ 
cation (Friedman & Quinn, 2006). Nevertheless, high- 
quality STEM education does not seem to be happening 
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at scale. Science education is not typically expected of 
programs in the way that art, music, and physical activity 
are. As noted in a 2008 study from the Coalition for Sci¬ 
ence After School (Chi, Freeman, & Lee, 2008) surveys 
of frontline staff have revealed significant obstacles for 
informal science education in afterschool, including lack 
of staff buy-in, comfort, or experience in science; insuf¬ 
ficient staff training; and a lack of materials. To address 
the STEM gap in expanded learning programs, expecta¬ 
tions of programs must change and frontline staff must 
be supported with professional development in STEM. 

A National Strategy to Build STEM 
Education Systems 

In an effort to prepare all children for post-secondary 
success and a lifetime of science-based learning, 
the Collaborative for Building After-School Systems 
(CBASS) and TASC, with generous support from the 
Noyce Foundation, have developed a national initiative 
to institutionalize engaging, inquiry-based STEM 
experiences in afterschool. In 2007, TASC set out to 
stimulate a culture shift among afterschool leaders and 
staff in order to increase the demand for and delivery of 
high-quality informal science education in New York City 
afterschool programs. This strategy, Frontiers in Urban 
Science Exploration (FUSE), employs a twofold systemic 
approach to bring about this culture shift and shape 
practice. First, a “grasstops” strategy, led by local out-of- 
school time (OST) intermediary organizations, engages 
leaders and staff of schools and afterschool programs, 
along with government officials, science organization 
leaders, policymakers, and funders, in building 
enthusiasm and capacity for inquiry-based STEM learning 
after school. Second, a “grassroots” strategy gives frontline 
afterschool staff and supervisors who do not have STEM 
backgrounds the content knowledge, instructional skills, 
and confidence to facilitate STEM activities effectively 
CBASS is expanding the New York City work of FUSE in 
six locations—Baltimore, Boston, Chicago, Oakland (CA), 
Palm Beach County, and Providence—to demonstrate 
the feasibility of a systemic strategy to advance STEM 
education and to identify promising practices to inform 
policy and practice nationally. As of the submission of this 
article, evaluations of the initiative had been conducted in 
New York City, Providence, and Oakland; therefore, we 
focus on those cities’ promising practices and grassroots 
outcomes. Evaluations for the remaining four cities are 
forthcoming. 

The FUSE strategy is designed to be both flexible 
enough to be effective across jurisdictions and focused 


enough to result in similar shared effects. The strategy 
builds on local assets while maintaining broad core 
elements to support program success. Core elements 
of afterschool STEM programs fall into two categories: 
program and system (Table 1). Program-level elements 
describe characteristics of high-quality afterschool 
science education, while system-level elements describe 
characteristics of well-coordinated systems that lead to 
improved quality, scale, and sustainability. 

Promising Approaches 

Intermediary OST organizations in the cities where 
FUSE has been implemented have tested approaches at 
the grassroots and grasstops levels to foster the mindset 
that frontline staff members, though not necessarily 
trained in STEM disciplines, can effectively facilitate 
informal science education. Though FUSE embraces a 
holistic system approach targeted to frontline staff and 
city leadership, 2010-2011 evaluation findings pointed 
to a correlation among strong gains in staff and youth 
outcomes and grassroots activities directed toward 
frontline staff. These findings are preliminary; our future 
evaluations will look more closely at the effect of the 
grasstops strategy on sustainability and on culture shifts 
at the program and city leadership levels. 

Here we focus on promising practices from the 
2010-2011 school year in New York, Providence, and 
Oakland that have helped contribute to positive staff and 
youth outcomes. The practices fall into three categories: 

• Providing experiential, sequential training opportunities 

• Assessing quality STEM programs 

• Engaging staff in peer learning communities 

Experiential, Sequential Training Opportunities 

When TASC set out to increase the amount of informal 
science education in New York City afterschool programs, 
it built on existing high-quality curricula rather than 
creating its own. TASC’s criteria for high-quality science 
curricula included that they: 

• Be inquiry-based and hands-on 

• Involve youth in higher-order thinking skills such as 
decision making, planning, problem solving, and re¬ 
flecting 

• Include opportunities for parental involvement 

• Provide chances for youth to learn about role models 

• Encourage youth to see themselves as learners 

• Use techniques appropriate for a variety of learning 
styles, with attention to the needs of underrepresented 
populations 

• Use affordable materials that are easy to find 
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PROGRAM LEVEL SYSTEM LEVEL 


DEVELOP 

STAFF 

To ensure continuity of skills and expertise 
from year to year, training and technical 
assistance are: 

• Ongoing: conducted in multiple sessions 
across the year with repeated observation 
and coaching 

• Differentiated: incorporating advanced 
activities to ensure skill improvement for 
returning participants 

• Cohort-based: involving multiple sites 
trained as a group 

• Delivered to teams: attended by supervisors 
and frontline staff from the same site 

UTILIZE 

COORDINATING 

ENTITY 

A coordinating agent, such as an intermediary, 
supports the development of the informal 
science education strategy by: 

• Leveraging resources 

• Vetting curricula 

• Organizing training 

• Disseminating promising practices 

• Fostering partnerships and collaborations 

• Evaluating quality and impact 

ENGAGE 

CROSS-SECTOR 

LEADERS 

To stimulate a culture shift about the 
importance of STEM in afterschool, leaders 
from community, school, informal science, 
and business sectors are engaged through 
convenings, alliances, and strategic planning. 

INTEGRATE 

HIGH-QUALITY 

CURRICULA 

High-quality curricula: 

• Are designed for afterschool 

• Are inquiry-based and grounded in fun 

• Involve familiar materials to make the case 
that science is part of our everyday lives 

• Promote equity among boys and girls and 
among students of varying abilities and 
ethnicities 

• Are evidence-based 

• Are affordable 

PROMOTE CO¬ 
INQUIRY 

Staff and students work side by side to 
explore and test assumptions. 


Table 1. FUSE Core Elements 


Be easy to implement for staff with no science back¬ 
ground 

Address national STEM standards 

Include a staff training component 

Provide appropriate content for a diverse urban audience 


TASC created a menu of STEM 
curriculum options each year, 
ensuring that the offerings included 
a range of age levels and a variety of 
STEM subjects. The menu included 
descriptions of each program, the 
appropriate age range, the dates of 
trainings, and any costs associated with 
implementing the curriculum. TASC 
required sites to fill out an application 
form and sign a memorandum of 
understanding that detailed their 
commitment. TASC worked with sites 
to identify appropriate curricula and 
support the delivery of the activities 
based on each site’s STEM readiness 
and goals. 

TASC then designed a series of 
experiential, sequential training ses¬ 
sions for staff to attend throughout 
the year. At each training, the TASC 
STEM team facilitated and modeled 
the curricula through hands-on activi¬ 
ties so that staff had the opportunity 
to engage in the activities themselves 
before implementing them with 
youth. Experiential training in spe¬ 
cific informal science curricula gives 
site coordinators and frontline staff 
the curriculum, hands-on materials, 
and coaching they need to implement 
science education. Site staff gener¬ 
ally attended in teams of at least two 
to ensure consistency of STEM pro¬ 
gramming from year to year. Ongoing 
trainings throughout the year allowed 
staff to reflect with peers on what 
worked and what didn’t and to refine 
the co-inquiry pedagogical approach. 

Peer Learning Communities 

In an effort to increase the capacity 
of afterschool providers to provide 
accessible, high-quality informal 
science education as well as to 
develop staff members’ confidence in 
facilitating STEM activities, partners in 
Oakland, California, developed an intensive peer learning 
community. Staff from 25 sites across the Oakland 
Unified School District attended monthly meetings 
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convened by staff from the district and from Techbridge, 
a nonprofit organization that provides STEM experiences 
for underserved youth. Topics included teaching inquiry- 
based science, promoting science career exploration, 
engaging families and the community, supporting equity 
in science programming, integrating role models, and 
scaffolding science material so that students build on 
their knowledge and skills over the course of the year. 

Over the past two years of experimenting with the 
learning community, Techbridge found that a session 
works best if it includes the following components: 
peer-to-peer sharing on challenges and best practices, 
hands-on modeling of an activity where participants can 
observe best practices being implemented, reflection 
about the rationale behind the practice, and time to 
adapt the strategy to participants’ afterschool programs. 

As an essential complement to the learning community, 
each participant is paired with a trainer for the entire year to 
receive ongoing support. Participants receive two coaching 
sessions during the school year, in the fall and spring. Each 
session includes an observation of the participant leading 
a science lesson followed by a debrief to identify areas for 
improvement and develop action plans. 

The learning community contributed to staff 
motivation and confidence in facilitating STEM activities. 
One participant reported, “I used to have a hard time 
putting my lessons together, but now, because of the 
Science Learning Community, I can transform a regular 
lesson into a science lesson.” Another added, “1 used to be 
afraid of teaching science. Now I feel more comfortable 
because of the Science Learning Community.” 


The Effects of FUSE Afterschool STEM Systems 

New York City, Providence, and Oakland each developed 
strategies and systems to support and train their frontline 
staff to deliver high-quality STEM activities. The evalua¬ 
tions focused on outcomes of these grassroots strategies 
during the 2010-2011 school year. 

Using self-reported data from staff and youth, 
we explored the effect of the FUSE program on staff 
members’ instructional confidence and on youths’ 
STEM-related knowledge, confidence, motivation, and 
interest. The evaluation sought to answer the following 
research questions: 

• Does training have an impact on staff outcomes? 

• Does program dosage have an impact on youth 
outcomes? 

• Does training staff have an impact on youth outcomes? 

Methodology 

Staff members were surveyed at the beginning and at the 
end of the school year using an adapted version of the 
Science Teaching Efficacy Belief Instrument, developed by 
Riggs and Enochs (1990) for the National Association for 
Research in Science Teaching. Surveys were administered 
to determine how effectively staff members felt that they 
could teach science in afterschool and how much of an effect 
they thought they could have on youths’ science learning. 
The instrument consists of two subscales: the instructional 
confidence score, which indicates how confident the staff 
member is in his or her ability to effectively teach science, 
and the personal impact score, which measures how much 
the staff member believes that his or her teaching can 
influence youths’ science learning 
(Bursal, 2008). Tests showed strong 
reliability for the instructional 
confidence subscale and moderately 
strong reliability for the personal 
impact scale while validity tests 
revealed all items were significantly 
and positively correlated (Riggs 
& Enochs, 1990). Additionally, 
data on staff training dosage were 
collected for New York City, but not 
for Providence and Oakland. 

Twice during the year, youth 
participants were asked about their 
STEM-related knowledge, confi¬ 
dence, motivation, and interest. 
Two measures were used to assess 
these domains. At the first mea¬ 
surement, participants completed 


STEM Quality Assessment 


Observational tools to support quality improvement of STEM 
programs are emerging. For example. Dr. Gil Noam of the Program 
in Education, Afterschool, and Resiliency (PEAR) led the development 
of the DOS tool. The Educational Equity Center at FHI 360 developed 
a quality assessment tool that adds dimensions of gender equity 
through the Great Science for Girls project. 

As part of the FUSE initiative, the Providence After School Alliance 
(PASA) worked with the David P. Weikart Center for Youth Program 
Quality to develop and conduct a preliminary validation of a new 
observational assessment for STEM-focused OST programming. Based 
on the Weikart Center's Youth Program Quality Assessment (YPQA), 
the STEM PQA consists of both observational and interview forms. 
PASA is now using the STEM PQA to observe STEM-focused programs 
and to coach instructors on how to improve quality. 
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Figure 1. NYC Instructional Confidence Scores by Previous STEM Experience 
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the Excited, Engaged and Interested 
Learner Survey (Common Instru¬ 
ment), which is being validated 
during the 2012-2013 school year 
by PEAR. The tool asks youth about 
their STEM habits, engagement, and 
career plans and about their feelings 
toward both in-school and out-of- 
school STEM. At the second mea¬ 
surement, youth again completed 
this survey as well as an adapted 
version of the Student Science At¬ 
titude Change tool (originally called 
Student Subjective Attitude Change 
Measures) developed by Stake and 
Mares (2001) of the University of 
Missouri-St. Louis. This scale cap¬ 
tures participants’ assessment of 
the degree to which the program brought about posi¬ 
tive change in their science motivation, confidence, and 
knowledge. The adapted version used a four-point scale, 
from “not at all” to “definitely,” on which students rated 
statements in the form “My experiences in the afterschool 
science program [led to an outcome].” Tests of reliability 
resulted in strong reliability for motivation and confidence 
and moderately strong reliability for the knowledge scale 
(Stake & Mares, 2001). Youth program participation data 
was also collected for Providence and Oakland, but not for 
New York City 

Findings 

Findings center on staff members’ beliefs about their 
confidence and efficacy and on 
youth participants’ assessments of 
changes in their STEM knowledge 
and attitudes. 

Staff Members 

One key finding was that FUSE par¬ 
ticipation built confidence among 
inexperienced STEM instructors. 

Before FUSE training, staff who 
had previous STEM experience 
scored significantly higher on the 
instructional confidence scale than 
did those with no previous STEM 
experience. This difference is con¬ 
sistent with research suggesting 
that experienced teachers have 
higher self-efficacy beliefs than do 


novice teachers (Angle & Moseley, 2009). After training 
and a year of experience, the difference disappeared. This 
finding suggests that, after participating in FUSE, inex¬ 
perienced staff caught up with their more experienced 
peers. Figure 1 shows instructional confidence scores for 
New York City and Figure 2 for Oakland. None of Provi¬ 
dence’s staff members had previous STEM experience, so 
there was no comparison group. 

One city’s evaluation found that training attendance 
and dosage were associated with increased instructional 
confidence or personal impact scores. In New York City, 
two groups of staff took the end-of-year survey: a program 
group of staff who attended training and a comparison 
group of staff who did not. New York City was the 


Figure 2. Oakland Instructional Confidence Scores by Previous STEM Experience 
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Figure 3. NYC Instructional Confidence and Personal Impact Scores by 
Training Attendance 
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only city to distribute surveys to staff members who did not 
participate in training. The program group had significantly 
higher post-program instructional confidence scores than did 
the comparison group, as shown in Figure 3. Personal impact 
scores were similar for both groups. 

New York outcomes also showed that the dosage of 
training affected personal impact scores. Figure 4 shows 
that staff who attended one to three training sessions had 
a mean decline of 2.0 points in personal impact, while staff 
who attended four or more sessions had a mean increase 


Figure 4. NYC Instructional Confidence and Personal Impact Score Change by 
Training Dosage 
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of 2.4 points. This between-group 
difference in personal impact scores is 
statistically significant; it suggests that 
greater depth of training helps staff to 
see themselves as having an important 
effect on youths’ STEM learning. 

Furthermore, youth science motiva¬ 
tion and science confidence were both 
positively correlated with staff training 
in New York City, the only site that col¬ 
lected data on training dosage. Flaving 
staff members attend more training was 
correlated with greater student motivation 
and confidence in science, as found in the 
Attitude Change survey. In afterschool 
programs that had staff members attend 
more FUSE trainings, youth reported 
more positive feelings about engaging in 
science as a result of their program experiences. The Excited, 
Engaged, and Interested Learning Survey also showed a 
relationship between staff training dosage and youth atti¬ 
tudes about science. The number of training sessions staff 
attended was significantly and positively correlated with 
youths’ agreement with such statements as, “I like to take 
things apart and learn more about them,” “I would like to 
have a science or computer job in the future,” “I get excited 
to hnd out I will be doing a science activity,” and “Science 
is one of my favorite subjects after school.” These findings 
support those from the Attitude 
Change survey, where staff train¬ 
ing was found to be signihcantly 
correlated with student motiva¬ 
tion and confidence. 


Youth Participants 

The post-participation surveys 
found a relationship between 
level of student exposure to 
STEM and self-reported science 
knowledge. In both Oakland 
(Figure 5) and Providence 
(Figure 6), science knowledge 
was signihcantly higher for 
youth who were exposed to 
STEM curricula for more than 
one month than for those who 
had less than one month’s worth 
of STEM activities. Changes in 
science confidence scores were 
also higher in both cities for 
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Figure 5. Oakland Student Attitude Change by Dosage 
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STEM experiences into 
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Figure 6. Providence Student Attitude Change by Dosage 
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* p < 0.05 ** Not significant at the 0.05 level, but a trend exists. 


students with more than one month of participation, 
though the differences were not significant at the 0.05 
level. Student participation data were not available for 
New York City. 

Furthermore, Oakland’s survey results showed 
significantly higher science motivation and science 
confidence scores for youth who were exposed to STEM 
curricula for three or more hours per week than for 
those who had fewer than three hours per week of STEM 
activities, as shown in Figure 7. The trends in Providence 
were in the opposite direction, though the findings there 
were not statistically significant. New York, again, did 
not provide student participation data. 


Outreach should empha¬ 
size that youth develop¬ 
ment experts can 
facilitate STEM co-inquiry. 

Outreach to afterschool 
programs and schools 
should aim to build 
public understanding 
that anyone with ap¬ 
propriate training and 
support—not just sci¬ 
ence experts—can 
implement STEM in 
afterschool programs. 
Successful informal sci¬ 
ence programs draw on 
the youth development 
expertise of afterschool 
leaders to adopt a co¬ 
inquiry approach, in 
which leaders learn 
alongside students. Broadening the understanding of who 
can deliver afterschool science education helps to build 
the case that afterschool is a natural place to engage 
young people in science. 

Curriculum matters. Selecting appropriate and high- 
quality curriculum materials is essential to providing 
youth with hands-on STEM experiences that engage 
and excite them. Activities should be relevant to the 
participants, inquiry-based, and hands-on. Curricula 
that use easy-to-access, culturally familiar materials send 
a powerful message to the participants that science is 
everywhere, giving them an opportunity to continue the 
learning beyond the afterschool setting. 
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Figure 7. Oakland Student Attitude Change by Intensity 


■ < 3 hrs/wk (N = 300) ■ 3 hrs/wk+ (N = 54) 
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* p < 0.05 ** Not significant at the 0.05 level, but a trend exists. 


Provide training to staff members in order to boost 
staff and youth outcomes. Attending training may help 
staff without STEM background or experience rise to 
the same levels of confidence as STEM-proficient staff 
members. Training sessions are particularly helpful when 
they are hands-on and ongoing, allowing staff members 
to “be the youth.” Staff members learn to anticipate youth 
questions and comments and, as a result of learning 
by doing, increase their confidence in their teaching. 
Training not only benefits staff members but also supports 
youth outcomes, ffaving more trained staff members at 
a site is correlated with higher student STEM confidence 
and motivation. In addition, continuous onsite coaching, 
in which quality advisors observe STEM activities and 
work with staff to identify areas for improvement and 
to develop action plans, supports program improvement. 

Ensure consistent and sustained STEM participation for 
youth. Young people enjoy inquiry-based STEM activities 
after school; they report that participation increases their 
knowledge about science. STEM activities should not be 
a special event in afterschool programs. Rather, students 
should have opportunities to engage in STEM activities 
regularly in order to build on what they’ve learned in 
previous sessions. FUSE evaluations found that sustained 
involvement correlated with youth reports that activities 
increased their science knowledge. Additionally, more 
intense exposure, such as three or more hours per week, 
correlated with youth reports that activities increased 
how much they cared about science and how confident 
they felt about their science abilities. These relationships 
demonstrate the important role that afterschool science 
education can play in transforming STEM learning for kids. 


Coordinating entities 
are important change 
agents in building 
quality informal science 
education systems. 
A lead coordinating 
agency, such as an in¬ 
termediary, helps to 
broker partnerships 
and has a bird’s-eye 
view of a community’s 
resources for support¬ 
ing STEM education. 
In line with their core 
functions, intermediar¬ 
ies can provide profes¬ 
sional development, 
leverage resources, convene stakeholders, and conduct 
research to expand and sustain afterschool systems that 
promote informal science education. Coordinating entities 
are well positioned to bring high-quality STEM to scale. 
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